Refrigerator with contents monitoring system

ABSTRACT

A refrigerating appliance for receiving containers within the refrigerating appliance for storing a substance includes a cabinet at least partially defining an interior with an access opening. A door is movably mounted to the cabinet to selectively open and close the access opening, the door at least partially defining the interior when the door selectively closes the access opening. At least one thermal imaging device is configured to obtain thermal image data of at least one container within the interior. A control unit is configured to receive the thermal image data.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation of U.S. patent application Ser. No.15/692,285, filed Aug. 31, 2017, and now allowed, which is incorporatedherein by reference in its entirety.

BACKGROUND

In a refrigerator or other appliance for storing containers of foodsubstances, which can be found within a kitchen environment, garages,bars, restaurants, and other places, there can be numerous containers ofvarious types, sizes, and shapes that are configured to store all matterof food substances. It can be useful to know the amount of a foodsubstance that is remaining within a container, in order to avoid foodwaste and in order to determine when it is or is not necessary topurchase additional quantities of a specific food substance. However,determining the amount of the food substance stored in the container canbe difficult to ascertain.

Previous methods of determining amounts of food substances stored withincontainers in a refrigerator include estimating remaining contents basedon the mass of the container, visual imaging of containers, and userinput regarding consumption of the food substance. Estimating thecontents of a container based on the mass requires many sensors to beprovided in order to measure the mass of different containers,increasing production costs and complexity of the refrigerator. Visualimaging can require the use of transparent containers and result ininaccurate measurements when opaque containers are present. Requiring auser to input information places an extra burden on the user which isnot desirable.

BRIEF SUMMARY

In one aspect, illustrative aspects in accordance with the presentdisclosure relate to a refrigerating appliance for receiving containerswithin the refrigerating appliance for storing a substance, therefrigerating appliance including a cabinet at least partially definingan interior with an access opening, a door movably mounted to thecabinet to selectively open and close the access opening and at leastpartially defining the interior when the door selectively closes theaccess opening, at least one thermal imaging device configured to obtainthermal image data of at least one container within the interior, and acontrol unit configured to receive the thermal image data and programmedto identify and define a filled portion and an unfilled portion of theat least one container by pixel counting of the thermal image data, toanalyze temperature differences of the filled portion and the unfilledportion of the at least one container from the thermal image data, tocompare the area of the filled portion or the unfilled portion to theoverall size of the at least one container, and to output an amountindicator indicative of an amount of the substance in the at least onecontainer.

In another aspect, illustrative aspects in accordance with the presentdisclosure relate to a refrigerating appliance for receiving containerswithin the refrigerating appliance for storing a substance, therefrigerating appliance including a cabinet at least partially definingan interior with an access opening, a door movably mounted to thecabinet to selectively open and close the access opening and at leastpartially defining the interior when the door selectively closes theaccess opening, at least one thermal imaging device configured to obtainthermal image data of at least one container within the interior, and acontrol unit configured to receive the thermal image data and programmedto identify and define a filled portion and an unfilled portion of theat least one container from the thermal image data, to analyzetemperature differences of the filled portion and the unfilled portionof the at least one container from the thermal image data, and to outputan amount indicator indicative of an amount of the substance in the atleast one container, wherein the control unit is further programmed toidentify when the temperature of the filled portion has met or exceededa predetermined temperature threshold based on the thermal image data.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings:

FIG. 1 illustrates a perspective partial view of a refrigeratingappliance in the form of a refrigerator according to an aspect of thepresent disclosure.

FIGS. 2A and 2B illustrate schematic representations of containers thatcan be placed within the refrigerating appliance of FIG. 1 in accordancewith the present disclosure.

FIG. 3 illustrates a flowchart depicting exemplary steps and proceduresrelated to a contents monitoring system for assessing the contents ofthe container of FIG. 2 in accordance with the present disclosure.

FIG. 4 illustrates a flowchart depicting exemplary steps and proceduresrelated to a contents monitoring system that provides notifications to auser based on the assessment of the contents as illustrated in FIG. 3 inaccordance with the present disclosure.

DETAILED DESCRIPTION

FIG. 1 illustrates a storage and consumption environment such as akitchen 10 that can include a refrigerating or food storage appliance,illustrated herein as a refrigerator 12. The refrigerator 12 includes acabinet 6 at least partially defining an interior 8 that can hold aplurality of containers 16. The cabinet 6 includes an access opening.The refrigerator 12 can also include a door 14 that is movably mountedto the cabinet 6 for movement between an opened condition and a closedcondition in order to selectively open or close the access opening andthat at least partially defines the interior 8 when the door 14selectively closes the access opening of the refrigerator 12. While therefrigerating appliance is illustrated as a side-by-side, front-openingrefrigerator 12, the aspects of the present disclosure can haveapplicability in other refrigerating appliances, non-limiting examplesof which include stacked style freezer-on-top or freezer-on-bottomrefrigerators, drawer-style refrigerators or freezers, beverage coolers,free-standing refrigerators, build-in refrigerators, displayrefrigerators, etc.

Containers 16 can be any sort of container for holding a food substance.Containers 16 can be commercially available containers that are obtainedby a user already containing a food substance, or storage containersinto which a food substance is placed by a user for refrigerated storageoutside of the original packaging. Containers 16 can be transparent oropaque, with any suitable level of opacity being contemplated. The foodsubstance within the containers 16 can be any food substance,non-limiting examples of which include liquids, solids, gelatinoussubstances, mixtures, dry goods, etc. In an exemplary aspect, containers16 are used to contain food substances that are non-solid, non-limitingexamples of which include milk, juices, ketchup, other condiments,mayonnaise, jellies, sauces, creams, etc.

The refrigerator 12 further comprises a control unit 18 that is operablycoupled with at least one thermal imaging device 20 for monitoring theamount of a food substance that is present in at least one container 16.The control unit 18 can communicate with the at least one thermalimaging device 20 in a wired or a wireless manner, or a combination ofboth. The control unit 18 can be programmed to instruct the at least onethermal imaging device 20 to take an image or images of the contents ofthe refrigerator 12 in response to the occurrence of a predeterminedtrigger event, non-limiting examples of which include the opening orclosing of the door 14, the door 14 remaining open for a predeterminedperiod of time, the removal or addition of one or more containers 16, auser input or request, or a timer based trigger event. Regularvariations in temperature within the refrigerator 12, even when the door14 has not been recently opened, can result in perceptible thermaldifferences within the containers 16, so it can be desirable for thecontrol unit 18 to instruct the thermal imaging device 20 to take animage or images after a predetermined amount of time has elapsed,regardless of whether or not the door 14 has been opened or closed. Thecontrol unit 18 can further be programmed by a computer programcomprising a set of executable instructions to receive thermal imagingdata from the at least one thermal imaging device 20, process thethermal imaging data by analyzing temperature differences of thecontainer 16 from the thermal imaging data, and infer information aboutthe containers 16 and the contents of the container 16, which can thenbe communicated to the user, including the output of an amount indicatorindicative of the amount of a food substance in the container. Thecontrol unit 18 that performs the thermal imaging data processing andanalysis can be a thermal imaging device controller, a refrigeratingappliance controller, a combination of both, or a cloud-basedcontroller.

Any suitable number of thermal imaging devices 20 can be provided withinthe refrigerator 12, from a single thermal imaging device 20 to aplurality of thermal imaging devices 20 that can be arranged throughoutthe refrigerator 12. The at least one thermal imaging device 20 can beplaced on the door of the refrigerator 12, on the side walls or the rearwall of the refrigerator 12, or in any suitable location either withinthe refrigerator 12 or located on the refrigerator 12 or the door 14such that the thermal imaging device 20 can obtain an image of thecontents of the refrigerator 12. The thermal imaging devices 20 can bepositioned such that they are aligned with a predetermined area withinthe refrigerator 12 in order to perform thermal imaging on onlycontainers 16 that are located within the predetermined area of therefrigerator 12, or the thermal imaging devices 20 can be positionedsuch that thermal imaging can be performed on containers 16 at anylocation throughout the refrigerator 12. In the case that only apredetermined area of the refrigerator 12 are capable of being thermallyimaged by the thermal imaging devices 20, non-limiting examples of sucha predetermined area can include a single shelf or specific shelves ofthe refrigerator 12, or only containers 16 that are located withinshelves located on the door 14 of the refrigerator 12. The predeterminedarea can be predefined to receive containers 16 having specific desiredcontents. The specific desired contents can be, by way of non-limitingexample, programmably defined either by the appliance or according to apreference of a user, such as by a user interface that allows a user todesignate an area as being associated with containers 16 having specificdesired contents.

The thermal imaging device 20 can be any suitable device capable ofdetecting a thermal profile or thermal inertia of the contents of acontainer 16. Examples of such a suitable thermal imaging device 20include thermal cameras, thermal sensors, or infrared thermal imagingdevices such as infrared cameras such as forward looking infrared (FLIR)cameras, or infrared sensor arrays.

FIG. 2A illustrates a schematic view of a container 16 that can beimaged by the thermal imaging device 20 in order to monitor the amountof food substance remaining within the container 16. A fill level 22represents the amount of the food substance present within the container16. The fill level 22 defines both a filled portion 26 of the container16, as well as an unfilled portion 24 of the container 16. In thecontainer 16 shown in FIG. 2A, the container 16 is nearly completelyfull.

FIG. 2B illustrates a schematic view of a container 16 that is onlypartially full of a food substance. The fill level 22 is lower thanshown in FIG. 2A, defining a smaller filled portion 26 and a largerunfilled portion 24 of the container 16, as compared to the container 16illustrates in FIG. 2B.

Turning now to the operation of the contents monitoring system whereinthe containers 16 are subject to thermal imaging by the thermal imagingdevice 20, FIG. 3 illustrates a flow chart outlining the steps ofoperation of the control unit 18 and the thermal imaging device 20.First, at step 50, a triggering event occurs. In an exemplary aspect,the triggering event can be the closing of the door 14 of therefrigerator 12, regardless of whether or not any containers 16 or othercontents of the refrigerator 12 are actually touched or moved. It isalso contemplated that the triggering event can be the activation of asensor that indicates the presence of at least one container 16 within apredetermined area of the refrigerator 12. Non-limiting examples of suchsensors can include a capacitive sensor to identify the presence of acontainer 16 or a visual imaging device that recognizes the presence orabsence of a container 16 and can identify when the container 16 hasbeen removed or replaced within the refrigerator 12. When the door 14 isclosed, the control unit 18 causes the thermal imaging device 20 to beactivated, at step 52, and to take a thermal image of at least a portionof the contents of the refrigerator 12, which can include at least onecontainer 16. The thermal imaging data captured in the thermal imagegenerated by the thermal imaging device 20 can then be analyzed, at step54. This analysis can be performed by the control unit 18, or can beperformed as part of the function of the thermal imaging device 20itself, or by a separate processor, wherein any of the options for theanalysis can be locally available as a part of the refrigerator 12, orcan be a cloud-based service. The analysis of the thermal image data canthen be used to determine the location of the fill level 22, at step 56,and thus to infer the amount of a food substance remaining within thecontainer 16.

This analysis and determination of amount of a food substance within thecontainer 16 is possible because the filled portion 26 and the unfilledportion 24 of the container 16 have differing thermal inertias. Thermalinertia is defined as the rate of change in temperature at which thetemperature of a substance approaches that of its surroundings. Thethermal inertia of a substance is dependent upon the absorptivity of thesubstance, the specific heat of the substance, the thermal conductivityof the substance, the dimensions and/or volume of the substance, andother factors. The food substance within the filled portion 26 and theair within the unfilled portion 24 have differing thermal inertias, andthus will change temperature at different rates when, for example, thedoor 14 of the refrigerator 12 is opened, or when the container 16 isremoved from the refrigerator 12 for a period of time. Even if the door14 is opened only for a few seconds, the temperature of the filledportion 26 and the unfilled portion 24 can differ by 1-2° C.

For example, if the door 14 of the refrigerator 12 is opened for severalseconds, and then closed, the thermal imaging device 20 is activated anda thermal image is obtained of the container 16 as shown in FIG. 2A. Theunfilled portion 24 could have a temperature of, for example, 8° C.,while the filled portion 26 can have a temperature of 6° C. The controlunit 18 can interpret the thermal image data and infer that the filllevel 22 is at the junction between the 6° C. filled portion 26 and the8° C. unfilled portion 24, and can subsequently register that thecontainer 16 is nearly full. The determination of the amount remainingin the container 16 can be presented, by way of non-limiting example, asa quantitative amount, such as percent full (i.e. 95% full) based on thepercentage of the visual area of the container 16 that is at a firsttemperature versus the percentage of the visual area of the container 16that is at a second, differing temperature from the first. Thedetermination of the amount remaining in the container 16 can also oralternately be presented as a qualitative amount such as, by way ofnon-limiting example, full, half-full, or nearly empty. It will also beunderstood that a user can set the indications to be received, and themanner in which they are presented, whether it be qualitative orquantitative. In the case of the container 16 illustrated in FIG. 2B,the control unit 18 would recognize that a smaller percentage of thecontainer 16 is at the lower temperature, indicating that the fill level22 of the container 16 in FIG. 2B is much lower than the fill level 22of the container 16 in FIG. 2A.

The determination of the amount remaining in the container 16 can beaccomplished, by way of non-limiting example, by determining a ratio ofthe filled portion 26 and the unfilled portion 24 or by comparing thesize of the filled portion 26 or the unfilled portion 24 to the overallsize of the container 16. In the case that the thermal imaging device 20is used to generate thermal imaging data of a predetermined area of therefrigerator 12 in which a container 16 of predetermined shape or volumeis placed, the area of the filled portion 26 or the unfilled portion 24can be compared to a known reference value of the size or volume of thecontainer 16 of predetermined shape or predetermined volume. The area ofthe filled portion 26 and/or the unfilled portion 24 can be determiningby a variety of thermal imaging data analysis techniques, including, byway of non-limiting example, pixel counting.

Once the control unit 18 has determined the amount of food substanceremaining in the container 16, which can be expressed, for example, as apercent fullness, the control unit 18 can further be programmed toreport to a user of the refrigerator 12 when the percent fullness hasmet or exceeded a predetermined threshold. For example, the control unit18 can cause a message to be displayed to a user, either on a display onthe refrigerator 12 or by communication with another electronic deviceaccessible by the user, non-limiting examples of which can include acomputer, mobile phone, tablet, e-mail account, or an electronic deviceapplication, such as one that generates a shopping list, that thepercent fullness for a particular container 16 has reached or is below,for example, 25% full, as described at steps 58, 60 in FIG. 4. Thepredetermined fullness threshold at which a user would be notified canbe set according to the preference of a user, as a user-definablethreshold, or the threshold can be defined by the refrigerator 12.

In addition to monitoring an amount or percent of a food substanceremaining in a container 16 and providing information to a user on theamount of the food substance remaining in the container 16, additionalinformation relating to the status of a container 16 and the contentscan be gathered from the use of the thermal imaging device 20 andresulting thermal imaging data, as illustrated in FIG. 4. For example,if the control unit 18 determines from the thermal imaging data receivedfrom the thermal imaging device 20 that the temperature of the filledportion 26 has met or exceeded a predetermined threshold at step 62, forexample because the container 16 was left outside of the refrigerator 12for too long before being placed back into the refrigerator 12, thecontrol unit 18 can determine that the contents of the container 16 havegotten too warm and may be spoiled and notify the user at step 64 of theundesirably high temperature of the container 16.

It is also contemplated that the contents monitoring system can beemployed along with a transactional contents monitoring feature, forexample that tracks when a specific container 16 was placed within therefrigerator 12, to provide further useful information to the user. Forexample, if the control unit 18 receives information that a specificcontainer 16 has been removed from the refrigerator 12 and anidentically shaped container 16 is replaced less than a minute later,but the new container 16 is 20° C. warmer than the container 16 that wasremoved from the refrigerator 12 less than a minute before, the controlunit 18 can infer that the newly placed container 16 must be a newcontainer 16, rather than the same container 16, because the temperaturechange is too great to have occurred during the time the container 16was removed from the refrigerator 12, as described at steps 66, 68.

In a household kitchen, particularly when children have access to thekitchen, it may be difficult to regulate or keep track of the removal offood substances from containers. In a commercial kitchen includingmultiple food preparers rapidly preparing dishes, the task of trackingthe amounts of food substances in numerous containers can be even morechallenging. A refrigerator with an improved system for monitoring thequantity of remaining food substance contents within containers in therefrigerator can save on production costs and complexity of therefrigerator, improve user experience with the appliance, and provide anumber of benefits for the user, such as facilitating more informed foodconsumption and food purchase decisions.

The refrigerator contents monitoring system as described herein allowsfor a user to easily ascertain the amount of a food substance remainingin a container, without the necessity of having specific sensorsassociated with each container. Moreover, the system can function bothwith commercial containers, as well as with temporary storage containersthat are filled by the user and placed within the refrigerator. Thesystem will also function properly regardless of whether the containersare transparent or opaque, and can be used to monitor containers eitherthat are placed in a specific position within the fridge, or can be usedto monitor containers throughout the body of the refrigerator. Inaddition, a great variety of different food substances within containerscan be monitored in this way. The system can be employed to notify auser when a quantity of a specific food substance is getting low, when acontainer with a food substance may have been left out too long and isno longer safe for consumption, and also to identify when an oldcontainer may have been replaced with a new container. Thus, a singletype of thermal imaging device can be used to support a variety offunctions to provide additional useful information to users of therefrigerator.

While the use of the thermal imaging device has been described hereinwithin the context of a refrigerating appliance, it will be understoodthat the disclosure is applicable to any appliance for the storage offood substances in which the temperature is regulated or stored at atemperature different from the external environment of the appliance,whether the appliance stores the food substances at a temperature thatis lower or higher than the external environment. Non-limiting examplesof such an appliance for the storage of food substances include astorage or refrigerating cabinet, a storage or refrigerating drawer, abeverage storing appliance, such as for wine, spirits, liqueurs, etc.,or a wine cellar.

To the extent not already described, the different features andstructures of the various aspects can be used in combination with eachother as desired. That one feature may not be illustrated in all of theaspects is not meant to be construed that it cannot be, but is done forbrevity of description. Thus, the various features of the differentaspects can be mixed and matched as desired to form new aspects, whetheror not the new aspects are expressly described. All combinations orpermutations of features described herein are covered by thisdisclosure.

While the present disclosure has been specifically described inconnection with certain specific aspects thereof, it is to be understoodthat this is by way of illustration and not of limitation. Reasonablevariation and modification are possible within the scope of the forgoingdisclosure and drawings without departing from the spirit of the presentdisclosure which is defined in the appended claims.

What is claimed is:
 1. A refrigerating appliance for receivingcontainers within the refrigerating appliance for storing a substance,the refrigerating appliance comprising: a cabinet at least partiallydefining an interior with an access opening; a door movably mounted tothe cabinet to selectively open and close the access opening and atleast partially defining the interior when the door selectively closesthe access opening; at least one thermal imaging device configured toobtain thermal image data of at least one container within the interior;and a control unit configured to receive the thermal image data andprogrammed to identify and define a filled portion and an unfilledportion of the at least one container by pixel counting of the thermalimage data, to analyze temperature differences of the filled portion andthe unfilled portion of the at least one container from the thermalimage data, to compare the area of the filled portion or the unfilledportion to the overall size of the at least one container, and to outputan amount indicator indicative of an amount of the substance in the atleast one container.
 2. The refrigerating appliance of claim 1 whereinthe at least one container has a predetermined shape or volume.
 3. Therefrigerating appliance of claim 2 wherein the area of the filledportion or the unfilled portion is compared to a known reference valueof the size or the volume of the at least one container having thepredetermined shape or volume.
 4. The refrigerating appliance of claim 3wherein the interior comprises a predetermined area that is predefinedto receive containers having the predetermined shape or volume.
 5. Therefrigerating appliance of claim 1 wherein the control unit is one of athermal imaging device controller or a refrigerating appliancecontroller.
 6. The refrigerating appliance of claim 1 wherein thethermal imaging device is an infrared thermal imaging device.
 7. Therefrigerating appliance of claim 1 wherein the control unit determines apercent fullness of the at least one container and provides anotification to a user when the percent fullness of the at least onecontainer is determined to be below a predetermined threshold.
 8. Therefrigerating appliance of claim 7 wherein the predetermined thresholdis a user-definable threshold or is a threshold defined by therefrigerating appliance.
 9. The refrigerating appliance of claim 1wherein the thermal imaging device is activated at least one of eachtime the door is moved between an opened condition and a closedcondition or when a predetermined amount of time has elapsed.
 10. Therefrigerating appliance of claim 1 wherein the refrigerating applianceis one of a refrigerator, a refrigerating cabinet, a refrigerateddrawer, a beverage refrigerating appliance, or a temperature-controlledwine cellar.
 11. A refrigerating appliance for receiving containerswithin the refrigerating appliance for storing a substance, therefrigerating appliance comprising: a cabinet at least partiallydefining an interior with an access opening; a door movably mounted tothe cabinet to selectively open and close the access opening and atleast partially defining the interior when the door selectively closesthe access opening; at least one thermal imaging device configured toobtain thermal image data of at least one container within the interior;and a control unit configured to receive the thermal image data andprogrammed to identify and define a filled portion and an unfilledportion of the at least one container from the thermal image data, toanalyze temperature differences of the filled portion and the unfilledportion of the at least one container from the thermal image data, andto output an amount indicator indicative of an amount of the substancein the at least one container; wherein the control unit is furtherprogrammed to identify when the temperature of the filled portion hasmet or exceeded a predetermined temperature threshold based on thethermal image data.
 12. The refrigerating appliance of claim 11 whereinthe control unit is configured to define the filled portion and theunfilled portion of the at least one container by pixel counting of thethermal image data.
 13. The refrigerating appliance of claim 11 whereinthe control unit is further configured to determine a ratio of thefilled portion and the unfilled portion or to compare the area of thefilled portion or the unfilled portion to the overall size of the atleast one container to output the amount indicator indicative of theamount of the substance in the at least one container.
 14. Therefrigerating appliance of claim 11 wherein the control unit provides anotification to a user when the temperature of the filled portion of theat least one container has met or exceeded the predetermined temperaturethreshold.
 15. The refrigerating appliance of claim 11 wherein thecontrol unit is one of a thermal imaging device controller or arefrigerating appliance controller.
 16. The refrigerating appliance ofclaim 11 wherein the thermal imaging device is an infrared thermalimaging device.
 17. The refrigerating appliance of claim 11 wherein thecontrol unit determines a percent fullness of the at least one containerand provides a notification to a user when the percent fullness of theat least one container is determined to be below a predetermined percentfullness threshold.
 18. The refrigerating appliance of claim 17 whereinthe predetermined percent fullness threshold is a user-definablethreshold or is a threshold defined by the refrigerating appliance. 19.The refrigerating appliance of claim 11 wherein the thermal imagingdevice is activated at least one of each time the door is moved betweenan opened condition and a closed condition or when a predeterminedamount of time has elapsed.
 20. The refrigerating appliance of claim 11wherein the refrigerating appliance is one of a refrigerator, arefrigerating cabinet, a refrigerated drawer, a beverage refrigeratingappliance, or a temperature-controlled wine cellar.